1. Field of the Invention
The present invention relates, generally, to tracking apparatus and methods and to augmented reality (AR) technology for integrating or augmenting real information such as actual or captured real-world images with virtual information such as images of computer-generated objects. More particularly, the invention relates to a fiducial-based tracker or to a fiducial-based means for registering the virtual-world information to the real-world information within an AR system.
2. Description of the Related Technology
AR technology allows a person to see or otherwise sense a computer-generated virtual world integrated with the real world. The “real world” is the environment that an observer can see, feel, hear, taste, or smell using the observer's own senses. The “virtual world” is defined as a generated environment stored in a storage medium or calculated using a processor. A tracker system within the AR technology registers the virtual world to the real world to integrate virtual and real information in a manner usable by the observer.
AR technology essentially “connects” a human user to a database or to a virtual or partially virtual environment using any combination of the human user's visual or aural or kinesthetic senses. Alternatively, AR technology can be viewed as connecting the partially virtual environment to the real environment as observed by the human user. AR technology allows the human user to perceive and be guided by processed database information that is integrated with the real world. Additionally, AR technology may enable a human to participate in an activity in a virtual world by translating the human user's movements or activity within a defined area or volume of the real world into the desired response in the virtual world.
Visual AR technology includes “Video See Through” AR technology and “Optical See Through” AR technology. Video See Through AR technology uses a camera to capture real world images and electronically adds or integrates the virtual images to create the augmented image. Optical See Through AR technology projects the virtual images on a see-through display, enabling a human user to see the projected image of the virtual object on the display and the images of the real world objects through the display.
By allowing a human to quickly, accurately and appropriately retrieve and apply virtual information to a problem, a task or situation in the real world, AR technology provides significant benefits in a society that is being inundated with information. For example, because of the quick development of many new technologies, the time required to train a person or to assemble and service complex products is increasing. New products are being quickly introduced into the market while many older generations of products are being concurrently used by the general public. An AR system can augment or enhance the real world view of the equipment with instructional text, drawings and diagrams to enable service professionals to quickly, accurately, and competently work on the equipment. There are many other examples of the benefits provided by the AR technology, including but not limited to the following examples. The invention can be used to improve the efficiency and quality of assembling or inspecting components such as televisions, radios, computers, and other components. Similarly, the invention can be used to improve the efficiency and quality of servicing or maintaining systems by superimposing text, pictures, drawings, schematics, and other information from maintenance manuals, for example, upon a real-world object. An AR system can be used to simulate real world activities, such as piloting aircraft or ships, and further can be used for instructional training, education, or rehearsals. Additionally, the invention can be used to create games or other entertainment, or to form an artistic medium by capturing and manipulating the artist's motions. Furthermore, the invention can be used to document head and eye responses to stimuli by tracking and recording the motion of either the user's head or the user's pupil after a stimuli has been introduced.
The AR system of the present invention has been used to fabricate wire harnesses for commercial airline planes. Each commercial airliner typically contains hundreds of wire bundles and hundreds of miles of wire. As shown in FIG. 3, the known art requires an assembly worker to bundle the wires around pegs formed on plywood boards by viewing printouts of wiring diagrams and transposing the information to the board. The known method for fabricating wire harnesses is a slow and tedious process that can result in expensive and time-consuming mistakes, and contributes to the billions of dollars that the airline industry loses because of production delays. The boards are specially designed for various wire harnesses, and must be stored. However, the wire harness fabrication AR system and embodiment of the present invention has been shown to reduce cycle time by fabricating the quality wire harnesses up to 50% faster than the current methods, and enables planes, particularly the wire harnesses, to be quickly designed and produced according to the special requirements of a customer.
An AR system should allow a human user to easily and naturally interact with objects in both the real and virtual world. Therefore, a visual AR system should: (1) quickly and accurately detect and track input data such as coordinate marks or fiducials; (2) quickly process the input data to determine the relative position and orientation between the user and the target objects and register the virtual world objects to the real world objects; and (3) quickly and smoothly integrate the virtual world objects with the real world objects either by displaying or projecting an image of the virtual world objects over the real world objects or by electronically combining an image of the virtual world objects with a captured imaged of the real world objects. Therefore, an AR system should have a small latency period and a quick update rate. “Latency” is defined as the period of time between the moment when the input data is captured and the moment that the augmented information is displayed or otherwise presented in usable form to the user. “Update rate” is defined as the frequency of refreshing the displayed or otherwise presented information that is processed from new input data. Humans perceive their environment with an essentially continuous update rate and zero latency period. Therefore, a human will be able to easily and naturally interact with an AR system that has an update rate and latency period approaching a human's natural capabilities; i.e., “real time” or “near real time.” An update rate of at least 60 Hertz and a latency period of 16 milliseconds or less is considered “near real time.”
Known trackers include magnetic-based, moving infrared beam optical-based, light source (LED) optical-based, ultrasonic-based, and mechanical-based trackers. Problems associated with these trackers generally include stray fields, noise, cost, insufficient accuracy, and limited mobility. Magnetic or moving infrared beam optical based trackers have been used in fighter aircraft and helicopters. These trackers accurately measure the user's pose with update rates of twenty Hertz or greater, but are complex systems that work within a very limited volume and severely limit the mobility of a user. Furthermore, magnetic-based trackers are sensitive to and their accuracy may be detrimentally affected by metallic objects in the vicinity of the sensors. Light source-based trackers use a camera to track various light source patterns, such as Light Emitting Diode (LED) arrays, that are placed on a surface of a target object. These light source-based trackers are effective in limited volumes and severely restrict the movement of the user. Additionally, these trackers have a limited accuracy and a latency greater than 100 milliseconds. Ultrasonic-based trackers are subject to noise interference in practical environments. The light source-based and ultrasonic-based trackers use “beacons” or “active” fiducials. An active fiducial generates a signal that is detectable by a sensor. Mechanical tracking systems, comprising a pantograph arm attached to a head mounted display, have been used in U.S. Army gunship helicopters. These mechanical-based trackers also severely restrict the mobility of the user.
Applicants' invention provides an AR system, including a fiducial-based tracker system, which is believed to constitute an improvement over existing technology.